<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta name="description" content="A Large-Scale Supervised Few-shot Relation Classification Dataset with State-of-the-Art Evaluation">
    <meta name="author" content="Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun">
    <meta name="license" content="MIT">
    <title>FewRel: A Large-Scale Supervised Few-shot Relation Classification Dataset with State-of-the-Art Evaluation</title>
    <!-- Bootstrap core CSS-->
    <link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
    <!-- Custom fonts for this template-->
    <link href="vendor/fontawesome-free/css/all.min.css" rel="stylesheet" type="text/css">
    <!-- Page level plugin CSS-->
    <link href="vendor/datatables/dataTables.bootstrap4.css" rel="stylesheet">
    <!-- Custom styles for this template-->
    <link href="css/sb-admin.css" rel="stylesheet">
    <script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default"></script>
  </head>

  <body id="page-top">
    <nav class="navbar navbar-expand navbar-dark static-top" style="background-color:rgb(00,75,151)">
		<button class="btn btn-link btn-sm text-white order-1 order-sm-0" id="sidebarToggle" href="#">
        	<i class="fas fa-bars"></i>
      	</button>
      <a class="navbar-brand mr-1" href="../fewrel.html">FewRel</a>
      <div class="ml-auto ">
      </div>
      <ul class="nav navbar-nav navbar-right">
          <li><a href="../fewrel.html" style="margin-right:20px">Home</a></li>
          <li><a href="../1/fewrel1.html" style="margin-right:20px">FewRel 1.0</a></li>
          <li><a href="../2/fewrel2_da.html" style="margin-right:20px; color: white">FewRel 2.0 Domain Adaptation</a></li>
          <li><a href="../2/fewrel2_nota.html">FewRel 2.0 None-of-the-Above Detection</a></li>
      </ul>
    </nav>



    <header class="text-white" style="height:320px;padding:0px;background-color:rgb(00,128,255);">
      <div class="text-center div-b" style="height:300px;padding-top:70px;">
        <h1 style="font-weight:400;">Welcome to <i style="font-size:54px;font-weight:600;">FewRel</i></h1>
        <hr/ style="border-color:rgb(255,255,255);width:80px;border-width:3px;">
        <p class="lead" style="color:rgba(255,255,255,1.0)">a <i style="font-size:22px;font-weight:600;">Few</i>-shot <i style="font-size:22px;font-weight:600;">Rel</i>ation classification dataset</p>
        <br/>
        <a class="btn btn-primary btn-xl js-scroll-trigger" href="https://www.aclweb.org/anthology/D19-1649.pdf" style="border-color:rgba(255,255,255,0.7);background-color:rgba(00,00,00,0.2)">Paper</a>
        <a class="btn btn-primary btn-xl js-scroll-trigger" href="https://github.com/thunlp/FewRel" style="background-color:rgba(00,00,00,0.2);border-color:rgba(255,255,255,0.7);margin-left:20px;">GitHub</a>
      </div>
    </header>


    <div id="wrapper">

      		<div id="content-wrapper">
				<div class="container-fluid">
					<div class="row">
						<div class="col-sm-3">
							<ol class="breadcrumb">
								<li class="breadcrumb-item active">What is FewRel?</li>
							</ol>
							<br/>

							<div class="row">
								<div class="col-sm-12 mb-6" style="text-align:justify;">
									FewRel is a <b>Few</b>-shot <b>Rel</b>ation classification dataset, which features 70, 000 natural language sentences expressing 100 relations annotated by crowdworkers.
                                    <br><br>
									Please refer to our EMNLP 2018 paper to learn more about FewRel.
									<div style="text-align: center; margin-top: 10px">	
										<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://www.aclweb.org/anthology/D18-1514.pdf" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">Han, Zhu, Yu, Wang, et al., 2018</a>
									</div>
                                    <br>
                                    We add two more challenging settings: few-shot domain adaptation (DA) and few-shot none-of-the-above detection (NOTA) in FewRel 2.0 dataset. Please refer to our EMNLP 2019 paper for more details.
									<div style="text-align: center; margin-top: 10px">	
										<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://www.aclweb.org/anthology/D19-1649.pdf" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">Gao et al., 2019</a>
									</div>
									<br>
									Baseline models and a series of toolkits are released in this repo:
									<div style="text-align: center;">	
										<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://github.com/thunlp/FewRel" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">thunlp/FewRel</a>
									</div>
								</div>
							</div>
							<br/>

							<ol class="breadcrumb">
							<li class="breadcrumb-item active">Get Started</li>
							</ol>
							<br/>

							<div class="row">
							<div class="col-sm-12 mb-6" style="text-align:justify;">
							It is fairly easy for you to get started with <b>FewRel</b>. First, you can download a copy of the dataset by the following links. The dataset is distributed under the <a href="http://creativecommons.org/licenses/by-sa/4.0/legalcode">CC BY-SA 4.0</a> license:
							<p>
							<div style="text-align: center;">
							<div style="text-align: center;">	
							<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://github.com/thunlp/FewRel/raw/master/data/train_wiki.json" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">[Training Set] (13.6 MB)</a>
							</div>	

							<br/>
							<div style="text-align: center;">	
							<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://github.com/thunlp/FewRel/raw/master/data/val_pubmed.json" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">[Validation Set] (422 KB)</a>
							</div>

							</div>
							</p>
							

							<p>
							To evaluate your model, please sample a pair of input and standard output file from the validation set,
							and compare your output with the standard one. The sampling scripts and evaluations scripts used in official evaluation are also accessible through the following work sheet.
							</p>
							<div style="text-align: center;">	
								<a class="btn btn-primary btn-xl js-scroll-trigger" href="https://worksheets.codalab.org/worksheets/0xecafe23758774db4835ca82ba66da30a/" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">Evaluation Tutorial</a>
							</div>
							<br/>
							Once you have built a model that works to your expectations on the validation set, you can submit it to get official scores on the hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. 
							Instructions to submit your model is here!
							<div style="text-align: center;">	
								<a class="btn btn-primary btn-xl js-scroll-trigger" href="fewrel/codalab.html" style="background-color:rgba(00,00,00,0.5);border-color:rgba(255,255,255,0.7);margin-left:20px;">Submission Tutorial</a>
							</div>

							</div>
							</div>
							<br/>

					

							<ol class="breadcrumb">
							<li class="breadcrumb-item active">Citation</li>
							</ol>    
							<br/>
							<div class="row">
							<div class="col-sm-12 mb-6" style="text-align:justify;">
								<pre>
@inproceedings{han-etal-2018-fewrel,
    title = "{F}ew{R}el: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation",
    author = "Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D18-1514",
    doi = "10.18653/v1/D18-1514",
    pages = "4803--4809"
}

@inproceedings{gao-etal-2019-fewrel,
    title = "{F}ew{R}el 2.0: Towards More Challenging Few-Shot Relation Classification",
    author = "Gao, Tianyu and Han, Xu and Zhu, Hao and Liu, Zhiyuan and Li, Peng and Sun, Maosong and Zhou, Jie",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1649",
    doi = "10.18653/v1/D19-1649",
    pages = "6251--6256"
}
								</pre>
							</div>
							</div>
							<br/>


						</div>
						<div class="col-sm-9">
								<ol class="breadcrumb">
									<li class="breadcrumb-item active">Leaderboard</li>
								</ol>
								<br/>
								<div class="row">
										<div class="col-sm-12 mb-6" style="text-align:center;">
											<table class="table">
													<thead class="thead-light">
															<tr>
																<th scope="col">#</th>
																<th scope="col" style="width:30%">Model</th>
                                                                <th scope="col">Domain Adaptation<br>5-way-1-shot</th>
                                                                <th scope="col">Domain Adaptation<br>5-way-5-shot</th>
                                                                <th scope="col">Domain Adaptation<br>10-way-1-shot</th>
                                                                <th scope="col">Domain Adaptation<br>10-way-5-shot</th>
															</tr>
														</thead>
														<tbody>
															<!-- Note: this is just an illustration now. Real data should be dynamically inserted here. -->
                                                            <!-- Note: add result here. -->
<tr><th scope="row"><p>1</p></th><td style="word-break:break-word;">Anonymous Pony<br><strong></strong><br>(Jan 17, 2020)</td><td>76.71</td><td>86.69</td><td>66.72</td><td>78.46</td></tr><tr><th scope="row"><p>2</p></th><td style="word-break:break-word;">Anonymous Python<br><strong></strong><br>(Dec 25, 2019)</td><td>66.41</td><td>83.52</td><td>51.85</td><td>73.60</td></tr><tr><th scope="row"><p>3</p></th><td style="word-break:break-word;">Anonymous Groundhog<br><strong></strong><br>(Dec 25, 2019)</td><td>67.23</td><td>82.09</td><td>54.32</td><td>71.01</td></tr><tr><th scope="row"><p>4</p></th><td style="word-break:break-word;">BERT-PAIR<a href="https://www.aclweb.org/anthology/D19-1649.pdf">[paper]</a><a href="https://github.com/thunlp/fewrel">[code]</a><br><strong>THUNLP, Tsinghua University</strong><br>(Nov 3, 2019)</td><td>67.41</td><td>78.57</td><td>54.89</td><td>66.85</td></tr><tr><th scope="row"><p>5</p></th><td style="word-break:break-word;">Proto-ADV (CNN)<a href="https://www.aclweb.org/anthology/D19-1649.pdf">[paper]</a><a href="https://github.com/thunlp/fewrel">[code]</a><br><strong>THUNLP, Tsinghua University</strong><br>(Nov 3, 2019)</td><td>42.21</td><td>58.71</td><td>28.91</td><td>44.35</td></tr><tr><th scope="row"><p>6</p></th><td style="word-break:break-word;">Proto-ADV (BERT)<a href="https://www.aclweb.org/anthology/D19-1649.pdf">[paper]</a><a href="https://github.com/thunlp/fewrel">[code]</a><br><strong>THUNLP, Tsinghua University</strong><br>(Nov 3, 2019)</td><td>41.90</td><td>54.74</td><td>27.36</td><td>37.40</td></tr><tr><th scope="row"><p>7</p></th><td style="word-break:break-word;">Proto-BERT<br><strong></strong><br>(Nov 3, 2019)</td><td>40.12</td><td>51.50</td><td>26.45</td><td>36.93</td></tr><tr><th scope="row"><p>8</p></th><td style="word-break:break-word;">Proto-CNN<br><strong></strong><br>(Nov 3, 2019)</td><td>35.09</td><td>49.37</td><td>22.98</td><td>35.22</td></tr>

                                                            <!-- Note: result part end. -->

														</tbody>
											</table>
										</div>
								</div>
								<br/>
						</div>
					</div>
				</div>
				<!-- footer copyright -->
				<div class="container-fluid">
					<div class="row">
						<footer class="sticky-footer" style="width:100%;background-color:rgb(00,75,151);color:rgb(255,255,255);">
								<div class="container my-auto">
									<div class="copyright text-center my-auto">	
										<span>Copyright © <a href="http://nlp.csai.tsinghua.edu.cn/site2/">THUNLP</a> 2018</span>
									</div>
								</div>
						</footer>
					</div>
				</div>
			</div>
		</div>

    <a class="scroll-to-top rounded" href="#page-top" style="background-color:rgba(0,0,0,0.3)">
      <i class="fas fa-angle-up"></i>
    </a>

    <!-- Bootstrap core JavaScript-->
    <script src="vendor/jquery/jquery.min.js"></script>
    <script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>

    <!-- Core plugin JavaScript-->
    <script src="vendor/jquery-easing/jquery.easing.min.js"></script>

    <!-- Page level plugin JavaScript-->
    <script src="vendor/chart.js/Chart.min.js"></script>
    <script src="vendor/datatables/jquery.dataTables.js"></script>
    <script src="vendor/datatables/dataTables.bootstrap4.js"></script>

    <!-- Custom scripts for all pages-->
    <script src="js/sb-admin.min.js"></script>

    <!-- Demo scripts for this page-->
    <script src="js/demo/datatables-demo.js"></script>
    <script src="js/demo/chart-area-demo.js"></script>

  </body>

</html>
